Substring Statistics
Identifieur interne : 000906 ( Main/Exploration ); précédent : 000905; suivant : 000907Substring Statistics
Auteurs : Kyoji Umemura [Japon] ; Kenneth Church [États-Unis]Source :
- Lecture Notes in Computer Science [ 0302-9743 ] ; 2009.
Abstract
Abstract: The goal of this work is to make it practical to compute corpus-based statistics for all substrings (ngrams). Anything you can do with words, we ought to be able to do with substrings. This paper will show how to compute many statistics of interest for all substrings (ngrams) in a large corpus. The method not only computes standard corpus frequency, freq, and document frequency, df, but generalizes naturally to compute, df k (str), the number of documents that mention the substring str at least k times. df k can be used to estimate the probability distribution of str across documents, as well as summary statistics of this distribution, e.g., mean, variance (and other moments), entropy and adaptation.
Url:
DOI: 10.1007/978-3-642-00382-0_5
Affiliations:
Links toward previous steps (curation, corpus...)
- to stream Istex, to step Corpus: 002855
- to stream Istex, to step Curation: 002658
- to stream Istex, to step Checkpoint: 000428
- to stream Main, to step Merge: 000914
- to stream Main, to step Curation: 000906
Le document en format XML
<record><TEI wicri:istexFullTextTei="biblStruct"><teiHeader><fileDesc><titleStmt><title xml:lang="en">Substring Statistics</title><author><name sortKey="Umemura, Kyoji" sort="Umemura, Kyoji" uniqKey="Umemura K" first="Kyoji" last="Umemura">Kyoji Umemura</name></author><author><name sortKey="Church, Kenneth" sort="Church, Kenneth" uniqKey="Church K" first="Kenneth" last="Church">Kenneth Church</name></author></titleStmt><publicationStmt><idno type="wicri:source">ISTEX</idno><idno type="RBID">ISTEX:5738EEDEE013A7676DE152BBFCAE3DEA695B4931</idno><date when="2009" year="2009">2009</date><idno type="doi">10.1007/978-3-642-00382-0_5</idno><idno type="url">https://api.istex.fr/document/5738EEDEE013A7676DE152BBFCAE3DEA695B4931/fulltext/pdf</idno><idno type="wicri:Area/Istex/Corpus">002855</idno><idno type="wicri:Area/Istex/Curation">002658</idno><idno type="wicri:Area/Istex/Checkpoint">000428</idno><idno type="wicri:Area/Main/Merge">000914</idno><idno type="wicri:Area/Main/Curation">000906</idno><idno type="wicri:Area/Main/Exploration">000906</idno></publicationStmt><sourceDesc><biblStruct><analytic><title level="a" type="main" xml:lang="en">Substring Statistics</title><author><name sortKey="Umemura, Kyoji" sort="Umemura, Kyoji" uniqKey="Umemura K" first="Kyoji" last="Umemura">Kyoji Umemura</name><affiliation wicri:level="1"><country xml:lang="fr">Japon</country><wicri:regionArea>Toyohashi University of Technology, Tempaku, 441-8580, Toyohashi, Aichi</wicri:regionArea><wicri:noRegion>Aichi</wicri:noRegion></affiliation></author><author><name sortKey="Church, Kenneth" sort="Church, Kenneth" uniqKey="Church K" first="Kenneth" last="Church">Kenneth Church</name><affiliation wicri:level="2"><country xml:lang="fr">États-Unis</country><wicri:regionArea>Microsoft, One Microsoft Way, 98052, Redmond, WA</wicri:regionArea><placeName><region type="state">Washington (État)</region></placeName></affiliation></author></analytic><monogr></monogr><series><title level="s">Lecture Notes in Computer Science</title><imprint><date>2009</date></imprint><idno type="ISSN">0302-9743</idno><idno type="eISSN">1611-3349</idno><idno type="ISSN">0302-9743</idno></series><idno type="istex">5738EEDEE013A7676DE152BBFCAE3DEA695B4931</idno><idno type="DOI">10.1007/978-3-642-00382-0_5</idno><idno type="ChapterID">5</idno><idno type="ChapterID">Chap5</idno></biblStruct></sourceDesc><seriesStmt><idno type="ISSN">0302-9743</idno></seriesStmt></fileDesc><profileDesc><textClass></textClass><langUsage><language ident="en">en</language></langUsage></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">Abstract: The goal of this work is to make it practical to compute corpus-based statistics for all substrings (ngrams). Anything you can do with words, we ought to be able to do with substrings. This paper will show how to compute many statistics of interest for all substrings (ngrams) in a large corpus. The method not only computes standard corpus frequency, freq, and document frequency, df, but generalizes naturally to compute, df k (str), the number of documents that mention the substring str at least k times. df k can be used to estimate the probability distribution of str across documents, as well as summary statistics of this distribution, e.g., mean, variance (and other moments), entropy and adaptation.</div></front></TEI><affiliations><list><country><li>Japon</li><li>États-Unis</li></country><region><li>Washington (État)</li></region></list><tree><country name="Japon"><noRegion><name sortKey="Umemura, Kyoji" sort="Umemura, Kyoji" uniqKey="Umemura K" first="Kyoji" last="Umemura">Kyoji Umemura</name></noRegion></country><country name="États-Unis"><region name="Washington (État)"><name sortKey="Church, Kenneth" sort="Church, Kenneth" uniqKey="Church K" first="Kenneth" last="Church">Kenneth Church</name></region></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/OcrV1/Data/Main/Exploration
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000906 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Exploration/biblio.hfd -nk 000906 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Ticri/CIDE
|area= OcrV1
|flux= Main
|étape= Exploration
|type= RBID
|clé= ISTEX:5738EEDEE013A7676DE152BBFCAE3DEA695B4931
|texte= Substring Statistics
}}
| This area was generated with Dilib version V0.6.32. |  |